Adaptive map-reduce pipeline with dynamic thread allocations

ABSTRACT

In an approach to adaptively pipeline a MapReduce job, a processor receives one or more data records from a storage and inserts the one or more data records into a map queue, wherein a size of the map queue is adaptive to one or more utilizations of one or more resources in the processor. One or more processors apply a map function to the one or more data records in the first buffer and sort the records that are output from the map function and store the sorted records. One or more processors receive and insert the sorted records into a reduce queue, wherein a size of the reduce queue is adaptive to one or more utilizations of resources in the one or more processors. One or more processors apply a reduce function to the sorted records in the reduce queue and store a result in a storage.

The following disclosure(s) are submitted under 35 U.S.C. 102(b)(1)(A)as prior disclosures by, or on behalf of, a sole inventor of the presentapplication or a joint inventor of the present application:

(i) “IBM Solution for Hadoop—Power Systems Edition”, An IBM ReferenceArchitecture for InfoSphere® BigInsights™, V1.0, Apr. 30, 2014, ©Copyright IBM Corporation 2014, pages 1-30.

(ii) “IBM Solution for Hadoop—Power Systems Edition”, An IBM ReferenceArchitecture for InfoSphere® BigInsights™, V2.0, Sep. 26, 2014, ©Copyright IBM Corporation 2014, pages 1-35.

BACKGROUND OF THE INVENTION

The present invention relates generally to the field of databasesystems, and more specifically to database systems that follow aMapReduce framework.

MapReduce is a programming model for processing large data sets, and thename of an implementation of the model by Google. MapReduce is typicallyused to do distributed computing on clusters of computers. The model isinspired by the “map” and “reduce” functions commonly used in functionalprogramming. MapReduce comprises a “Map” step wherein the master nodeestablishes a division of a problem in map tasks that each handle aparticular sub-problem and assigns these map tasks to worker nodes. Forthis, a scheduling master splits the problem input data and assigns eachinput data part to a map task. An input part is often referred to as asplit. The worker nodes process the sub-problems according to a map( )function provided by a user, and notify the master node upon map taskcompletion. MapReduce further comprises a “Reduce” step wherein themaster node assigns a “reduce” operation to some worker nodes, whichcollect the answers to all the sub-problems and analyze them, using areduce( ) function provided by the user, to form the output—the answerto the problem it was originally trying to solve.

MapReduce allows for distributed processing of the map and reductionoperations. Provided each mapping operation is independent of theothers, the maps can be performed in parallel. Similarly, a set of‘reducers’ can perform the reduction phase. While this process canappear inefficient compared to algorithms that are more sequential,MapReduce can be applied to significantly larger datasets than“commodity” servers can handle—a large server farm can use MapReduce tosort a petabyte of data in only a few hours; MapReduce is typicallysuited for the handling of ‘big data’. The parallelism also offers somepossibility of recovering from partial failure of servers or storageduring the operation: if one mapper or reducer fails, the work can berescheduled—assuming the input data is still available.

A significant design challenge associated with large complex systemsthat run MapReduce jobs is the efficient utilization of systemresources, principally CPU cycles and memory, on a spectrum of jobs thatvary greatly in their size and nature.

SUMMARY

Aspects of an embodiment of the present invention disclose a pluralityof methods for adaptively pipelining a MapReduce job. The methodincludes receiving, by a processor, into a first memory buffer, one ormore data records from a storage, wherein a size of the first memorybuffer is adaptive to one or more utilizations of one or more resourcesin the processor. The method further includes, inserting, by aprocessor, the first memory buffer into a first queue, wherein a size ofthe first queue is adaptive to one or more utilizations of one or moreresources in the processor. The method further includes, generating, byone or more processors, one or more output records from the first memorybuffer in the first queue by applying a map function to the one or moredata records in the first memory buffer. The method further includes,writing, by a processor, the one or more output records into a secondmemory buffer in a second queue, wherein the size of the second memorybuffer and the size of the second queue is adaptive to one or moreutilizations of one or more resources in the processor. The methodfurther includes, deleting, by one or more processors, the first memorybuffer from the first queue. The method further includes, generating, byone or more processors, one or more sorted records in the second memorybuffer by sorting the one or more output records in the second memorybuffer. The method further includes, writing, by a processors, thesecond memory buffer into a third queue, wherein the size of the thirdqueue is adaptive to one or more utilizations of one or more resourcesin the processor. The method further includes, deleting, by one or moreprocessors, the second memory buffer from the second queue. The methodfurther includes, merging, by one or more processors, one or more sortedrecords in the second memory buffer into an output data file that isstored in storage. The method further includes, deleting, by one or moreprocessors, the second memory buffer from the third queue. The methodfurther includes, generating, by one or more processors. The methodfurther includes, receiving, by one or more processors, an input datafrom a map stage of a MapReduce job into a memory region. The methodfurther includes, inserting, by a processor, the input data in thememory region into one or more third memory buffers in a fourth queue,wherein a size of a third memory buffer in the one or more third memorybuffers and the size of the fourth queue is adaptive to one or moreutilizations of resources in the processor. The method further includes,generating, by one or more processors, one or more output records byapplying a reduce function to an input data in a third memory buffer inthe fourth queue. The method further includes, inserting, by aprocessor, the one or more output records into a fourth memory buffer ina fifth queue, wherein the size of the fourth memory buffer and the sizeof the fifth queue is adaptive to one or more utilizations of the one ormore resources in the processor. The method further includes, writing,by one or more processors, the one or more output records in a fourthmemory buffer in the fifth queue to storage. The method furtherincludes, deleting, by one or more processors, the fourth memory bufferfrom the fifth queue.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 depicts a block diagram of a portion of a computing complex, inaccordance with an embodiment of the present invention.

FIG. 2 depicts a block diagram of node_0 depicted in FIG. 1, inaccordance with an embodiment of the present invention.

FIG. 3 depicts a detail of the map pipeline depicted in FIG. 2, inaccordance with an embodiment of the present invention.

FIG. 4 depicts a flowchart of the operational steps of the map pipelinedepicted in FIG. 3, in accordance with an embodiment of the presentinvention.

FIG. 5 depicts a detail of the reduce pipeline depicted in FIG. 2, inaccordance with an embodiment of the present invention.

FIG. 6 depicts a flowchart of the operational steps of the reducepipeline depicted in FIG. 5, in accordance with an embodiment of thepresent invention in FIG. 3, in accordance with an embodiment of thepresent invention.

FIG. 7 depicts a block diagram of node_0 that incorporates the mappipeline depicted in FIG. 3 and the reduce pipeline depicted in FIG. 5,in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION

Detailed embodiments of the present invention are disclosed herein withreference to the accompanying drawings. It is to be understood that thedisclosed embodiments are merely illustrative of potential embodimentsof the present invention and may take various forms. In addition, eachof the examples given in connection with the various embodiments isintended to be illustrative, and not restrictive. Further, the figuresare not necessarily to scale, some features may be exaggerated to showdetails of particular components. Therefore, specific structural andfunctional details disclosed herein are not to be interpreted aslimiting, but merely as a representative basis for teaching one skilledin the art to variously employ the present invention.

References in the specification to “one embodiment”, “an embodiment”,“an example embodiment”, etc., indicate that the embodiment describedmay include a particular feature, structure, or characteristic, butevery embodiment may not necessarily include the particular feature,structure, or characteristic. Moreover, such phrases are not necessarilyreferring to the same embodiment. Further, when a particular feature,structure, or characteristic is described in connection with anembodiment, it is submitted that it is within the knowledge of oneskilled in the art to affect such feature, structure, or characteristicin connection with other embodiments whether or not explicitlydescribed.

Embodiments of the present invention recognize that MapReduce jobs varygreatly in size and nature and that to efficiently utilize thecomputational resources of a map reduce system, appropriate sized unitsof work must be created and mapped to multiple threads of execution toexecute the units of work in parallel. Additionally, the size of thedata structures (e.g., memory buffers and queues) used in the flow ofwork and data in a MapReduce job must be dynamically adaptive to thenature of the work (e.g., CPU intensive, memory intensive, or storageintensive) and to the utilizations of various computer resources in themap reduce system.

FIG. 1 depicts system 100 that, in some scenarios and embodiments,includes client host 101, network 102, local disk 109, and map reducesystem 103, in which client host 101 submits one or more MapReduce jobsover network 102 to map reduce system 103. In an embodiment, map reducesystem 103 is comprised of master node 104, and n+1 worker nodes: node_0105, node_1 106, and additional nodes through node_N 107. Master node104 receives a job from client host 101 and partitions the job into maptasks, called splits, each map task handling a particular sub-problem,and assigns these map tasks to worker nodes node_0 105 through node_N107. Master node 104, and node_0 105 through node_N 107 are connected todistributed file system 108 and to local disk 109. Master node 104receives a MapReduce job from client 110 on client host 101 andpartitions the job and distributes the partitions of the job to thecomputer nodes.

FIG. 2 depicts node_0 105 in detail. In some scenarios and embodiments,node_0 105 is comprised of processors 201 and memory 202. Memory 202contains a software code and data structures that perform a MapReducejob: map pipeline 203, reduce pipeline 204, input data 205, intermediatedata 206, and output data 207. Processors 201 access the software codeand data structures in memory 202. Map pipeline 203 processes input data205 that is fetched from local disk 109, processes input data 205, andproduces intermediate data 206. Reduce pipeline 204 fetches intermediatedata 206, processes it, and output data 207 writes the output dataresult to distributed file system 108.

FIG. 3 depicts a detail of map pipeline 203. In some scenarios andembodiments, map pipeline 203 is comprised of decompression 309, memorybuffer 301, input record queue 302, map 303, memory buffer 304, outputrecord queue 305, sort 306, sorted output memory buffer 307, sortedoutput record queue 308, and merge 309 software functions. One or moreinput data records in local disk 109 are read into input data 205 inmemory 202. The data records in input data 205 are transferred intomemory buffer 301. If the data records in input data 205 are compressed,they are first decompressed in decompression 309 and then transferred tomemory buffer 301. Memory buffer 301 is then inserted into the tail ofinput record queue 302. Input record queue 302 is a first-in-first-outqueue. Map 303 processes the input records in the memory buffers ininput record queue 302 one at a time. Map 303 takes a memory buffer atthe head of output record queue 305 off the queue and processes theinput records in the memory buffer. Map 303 applies a map( ) functionthat is provided by client 110 on client host 101 to the input recordsin the memory buffer that it takes off of the head of input record queue302.

In some scenarios and embodiments, the size of memory buffer 301, andtherefore the amount of data records that memory buffer 301 contains, isadaptively controlled to enhance the utilization of CPU resources innode_0 105 by decreasing the I/O time to fetch the data records fromdistributed file system 108. The size of memory buffer 301 is adjusteddownward (i.e., decreased in size from large to small) until theprocessing of the data records in memory buffer 301 by map 303 beginbefore the processing of the amount data records in memory buffer 301would otherwise begin had memory buffer 301 been larger and theutilization of CPU resources is enhanced. Map 303 can begin processingthe data records in a smaller memory buffer 301 before map 303 can beginprocessing the data records in a larger memory buffer 301 because thesmaller memory buffer 301 can be filled faster than the larger memorybuffer 301 and therefore be inserted into input record queue 302 beforea larger memory buffer 301. Because map 303 processes fewer inputrecords in a smaller memory buffer 301 (when map 303 takes memory buffer303 from input record queue 302), the processing by map 303 willcomplete sooner that the processing would have completed on a greaternumber of input records in a larger memory buffer 303.

In some scenarios and embodiments, the size of input record queue 302(i.e., the number of memory buffers in it) is adaptively controlled toenhance CPU utilization by dynamically applying more compute threads tothe processing by map 303 when the size of input record queue 302 growsbeyond one or more specified size limits. Additionally, the number ofcompute threads applied to the processing by map 303 is dynamicallydecreased when the size of input record queue 302 decreases below one ormore specified size limits to free the compute threads for other work.

In some scenarios and embodiments, map 303 generates one or more outputrecords from the input records that map 303 receives from input recordqueue 302 and inserts the output records into memory buffer 304, whichis inserted into the tail end of output record queue 305. In somescenarios and embodiments, the size of memory buffer 304, and thereforethe amount of output data records that memory buffer 304 contains, isadaptively controlled to enhance the utilization of CPU resources innode_0 105. Output record queue 305 is a first-in-first-out queue thatholds one or more memory buffers inserted by map 303. Each memory buffercontains one or more output records produced by a map( ) functiondefined by client 110 and applied to input records from input recordqueue 302.

In some scenarios and embodiments, sort 306 processes the output recordsin the memory buffers in output record queue 305 one at a time. Sort 306takes a memory buffer at the head of output record queue 305 off thequeue and processes the output records in the memory buffer. Sort 306sorts the output records in the memory buffer and inserts the sortedoutput in sorted output memory buffer 307. Sorted output memory buffer307 is inserted into the tail of sorted output record queue 308.

In some scenarios and embodiments, the size of output record queue 305(i.e., the number of memory buffers in it) is adaptively controlled toenhance CPU utilization by dynamically applying more compute threads tothe processing by sort 306 when the size of output record queue 305grows beyond one or more specified size limits. Additionally, the numberof compute threads applied to the processing by sort 306 is dynamicallydecreased when the size of output record queue 305 decreases below oneor more specified size limits to free the compute threads for otherwork.

In some scenarios and embodiments, merge 309 takes one or more outputdata records from sorted output record queue 308, merges the one or moreoutput data records into a consistent, complete record of intermediateoutput data and inserts the record of intermediate output data intointermediate output data 206. Intermediate output data 206 temporarilyholds the record of intermediate output data. If the size of the recordof intermediate output data is larger than a specified threshold,intermediate output data 206 compresses the record of intermediateoutput data. If reduce pipeline 204 is not ready to accept the record ofintermediate output data, intermediate output data 206 stores the recordof intermediate output data in local disk 109. If reduce pipeline 204 isready to accept the record of intermediate output data, reduce pipeline204 reads the record of intermediate output data from intermediateoutput data 206.

In some scenarios and embodiments, reduce pipeline 204 is not in thesame compute node (e.g., node_0 105) that map pipeline 203 is in. Inthis case, intermediate output data 206 stores the record ofintermediate output data in local disk 109 and then reduce pipeline 204reads the record of intermediate output data from local disk 109 whenreduce pipeline 204 is ready to process the record of intermediateoutput data.

FIG. 4 depicts the operational steps performed by map pipeline 203, inan embodiment and scenario. Map pipeline 203 reads an input record (step402) and decides if the input record must be decompressed in decisionstep 404. In decision step 404, if the input record must be decompressed(decision step 404, YES branch), then the input record is decompressedby decompression 309 (step 406) and put into memory buffer 301 (step408). In decision step 404, if the input record does not have to bedecompressed (decision step 404, NO branch), then the input record putinto memory buffer 301 (step 408). Memory buffer 301 is put into inputrecord queue 302 (step 410). Map 303 removes an input record from inputrecord queue 302 and processes the input record with a map( ) functionthat is provided by client 110 (step 412) and included in the MapReducejob.

Map 303 produces an output record from the input record and inserts theoutput record into memory buffer 304 (step 414) and inserts memorybuffer 304 into output record queue 305 (step 416). Sort 306 removes oneor more memory buffers from output record queue 305, sorts the outputdata records in the one or more memory buffers (step 418), inserts thesorted output records into sorted output memory buffer 307 (step 420),and inserts sorted output memory buffer 307 into sorted output memoryqueue 308 (step 422). Merge 309 takes one or more output data recordsfrom sorted output record queue 308, merges the one or more output datarecords into a consistent, complete record of intermediate output dataand inserts the record of intermediate output data into intermediateoutput data 206 (step 424). Intermediate output data 206 writesintermediate output data to storage by storing the record ofintermediate output data in local disk 109 (step 426) and the processingof the input record by map pipeline 203 terminates (step 428).

FIG. 5 depicts a detail of reduce pipeline 204. In some scenarios andembodiments, reduce pipeline 204 is comprised of consolidation memorysegment 502, decompression 509, memory buffer 503, data record queue504, reduce 505, output memory buffer 506, output records queue 507, andcompressor 508. In an embodiment, reduce pipeline 204 and map pipeline203 are both in node_0 105. In this case, intermediate output data ispassed from map pipeline 203 to reduce pipeline within memory 202through intermediate data 206. If intermediate data 206 contains one ormore records of intermediate data produced by map pipeline 203,consolidation memory segment 502 fetches the one or more records andconsolidates records that are related to the same reduce task into aninput record. If the records of intermediate data are compressed,consolidation memory segment 502 decompresses the records withdecompression 509. Consolidation memory segment 502 consolidates the oneor more records that are related to the same map task together into aninput record and inserts the input record into memory buffer 503.

In an embodiment, reduce pipeline 204 and map pipeline 203 are not inthe same compute node. In this case, consolidation memory segment 502fetches the one or more records of intermediate output data produced bymap pipeline 203 from local disk 109 and, if the one or more records arecompressed, decompresses the one or more records with decompression 509.Consolidation memory segment 502 consolidates the one or more recordsthat are related to the same map task together into an input record andinserts the input record into memory buffer 503.

If map pipeline 203 and reduce pipeline 204 are both in node_105, thesize of related data consolidator 501, and therefore the amount of inputdata records that related data consolidator 501 can contain, isadaptively controlled to enhance the utilization of CPU resources innode_0 105 by decreasing the I/O time to fetch the data records fromlocal disk 109. Consolidation memory segment 502 consolidates the one ormore records that are related to the same reduce task together into aninput record and inserts the input record into memory buffer 503.

In some scenarios and embodiments, the size of consolidation memorysegment 502, and therefore the amount of data records that consolidationmemory segment 502 can contain, is adaptively controlled to enhance theutilization of CPU resources in node_0 105 by decreasing the I/O time tofetch the data records from local disk 109.

Memory buffer 503 is inserted into the tail of data record queue 504.Data record queue 504 is a first-in-first-out queue. Reduce 505processes the input records in the memory buffers in data record queue504 one at a time. Reduce 505 takes a memory buffer at the head of datarecord queue 504 off the queue and processes the input records in thememory buffer. Reduce 505 applies a reduce( ) function that is providedby client 110 on client host 101 to the data records in the memorybuffer that reduce 505 takes from the head of data record queue 504.

In some scenarios and embodiments, the size of memory buffer 503, andtherefore the amount of data records that memory buffer 503 contains, isadaptively controlled to enhance the utilization of CPU resources innode_0 105. The size of memory buffer 503 is adjusted downward (i.e.,decreased in size from large to small) until the processing of the ofthe data records in memory buffer 503 by reduce 505 begin before theprocessing of the amount data records in memory buffer 503 wouldotherwise begin had memory buffer 503 been larger and the utilization ofCPU resources is enhanced. Reduce 505 can begin processing the datarecords in a smaller memory buffer 503 before reduce 505 can beginprocessing the data records in a larger memory buffer 503 because thesmaller memory buffer 503 can be filled faster than the larger memorybuffer 503 and therefore be inserted into data record queue 504 before alarger memory buffer 503. Because reduce processes fewer data records ina smaller memory buffer 503 (when reduce 505 takes memory buffer 503from data record queue 504), the processing by reduce 505 will completesooner that the processing would have completed on a greater number ofinput records in a larger memory buffer 503.

In some scenarios and embodiments, the size of data record queue 504(i.e., the number of memory buffers in it) is adaptively controlled toenhance CPU utilization by dynamically applying more compute threads tothe processing by reduce 505 when the size of data record queue 504grows beyond one or more specified size limits. Additionally, the numberof compute threads applied to the processing by reduce 505 isdynamically decreased when the size of data record queue 504 decreasesbelow one or more specified size limits to free the compute threads forother work.

In some scenarios and embodiments, reduce 505 generates one or moreoutput records from the data records that reduce 505 receives from datarecord queue 504 and inserts the output records into output memorybuffer 506, which is inserted into the tail end of output records queue507. In some scenarios and embodiments, the size of output memory buffer506, and therefore the amount of output data records that output memorybuffer 506 contains, is adaptively controlled to enhance the utilizationof CPU resources in node_0 105. Output records queue 507 is afirst-in-first-out queue that holds one or more output memory buffers(e.g., output memory buffer 506) inserted by reduce 505. Each outputmemory buffer (e.g., output memory buffer 506) contains one or moreoutput records produced by a reduce( ) function defined by client 110and applied to data records from data record queue 504. One or moreoutput memory buffers that contain the result of the MapReduce operationare extracted from the head of output records queue 507 by output data207 and stored in distributed file system 108.

FIG. 6 depicts the operational steps performed by reduce pipeline 204,in an embodiment and scenario. Consolidation memory segment 502 inreduce pipeline 204 reads intermediate data (step 602) and decides ifintermediate data must be decompressed (decision step 604). In decisionstep 604, if the intermediate data must be decompressed (decision step604, YES branch), then the input record is decompressed by decompression509 (step 606). In decision step 604, if the intermediate data does nothave to be decompressed (decision step 604, NO branch), then the inputrecord is not decompressed by decompression 509.

Consolidation memory segment 502 consolidates the one or more recordsthat are related to the same map task together into an input record(step 608) and inserts the input record into memory buffer 503 (step610). Memory buffer 503 is inserted into the tail of data record queue504 (step 612). Reduce 505 extracts a memory buffer from the head ofdata record queue 504 (step 614) and applies the reduce function reduce() to the data records in the memory buffer (step 616). Reduce 505 writesthe output data records that reduce 505 generates from applying reduce() to the data records to output memory buffer 506 (step 618). Outputmemory buffer 506 is inserted into the tail of output records queue 507(step 620). The output memory buffer at the head of output recordsmemory buffer 507 is extracted (step 622) and if necessary (decisionstep 624, YES branch), is compressed (step 625) by compressor 508 and isnot compressed if compression is not necessary (decision step 624, NObranch). The output memory buffer at the head of the output recordsqueue 507 is then stored into distributed file system 103 by output data207 (step 207). The MapReduce operation terminates (step 628).

FIG. 7 depicts an exemplary embodiment of node_0 105, which, in anembodiment, hosts map pipeline 203 and reduce pipeline 204. Node_0 105includes processors 704 (which are processors 201), cache 716, andcommunications fabric 702, which provides communications between cache716, memory 706, persistent storage 708, communications unit 710, andinput/output (I/O) interface(s) 712. Communications fabric 702 can beimplemented with any architecture designed for passing data and/orcontrol information between processors (such as microprocessors,communications and network processors, etc.), system memory, peripheraldevices, and any other hardware components within a system. For example,communications fabric 702 can be implemented with one or more buses.

Memory 706 (which is memory 202) and persistent storage 708 are computerreadable storage media. In this embodiment, memory 706 includes randomaccess memory (RAM). In an embodiment, memory 706 contains map pipeline203, reduce pipeline 204, input data 205, intermediate data 206, andoutput data 207. In general, memory 706 can include any suitablevolatile or non-volatile computer readable storage media. Cache 716 is afast memory that enhances the performance of processors 704 by holdingrecently accessed data and data near accessed data from memory 706.

Program instructions and data used to practice embodiments of thepresent invention may be stored in persistent storage 708 for executionby one or more of the respective processors 704 via cache 716 and one ormore memories of memory 706. In an embodiment, persistent storage 708includes a magnetic hard disk drive. Alternatively, or in addition to amagnetic hard disk drive, persistent storage 708 can include a solidstate hard drive, a semiconductor storage device, read-only memory(ROM), erasable programmable read-only memory (EPROM), flash memory, orany other computer readable storage media that is capable of storingprogram instructions or digital information.

The media used by persistent storage 708 may also be removable. Forexample, a removable hard drive may be used for persistent storage 708.Other examples include optical and magnetic disks, thumb drives, andsmart cards that are inserted into a drive for transfer onto anothercomputer readable storage medium that is also part of persistent storage708.

Communications unit 710, in these examples, provides for communicationswith other data processing systems or devices. In these examples,communications unit 710 includes one or more network interface cards.Communications unit 710 may provide communications through the use ofeither or both physical and wireless communications links. Programinstructions and data used to practice embodiments of the presentinvention may be downloaded to persistent storage 708 throughcommunications unit 710.

I/O interface(s) 712 allows for input and output of data with otherdevices that may be connected to each computer system. For example, I/Ointerface 712 may provide a connection to external devices 718 such as akeyboard, keypad, a touch screen, and/or some other suitable inputdevice. External devices 718 can also include portable computer readablestorage media such as, for example, thumb drives, portable optical ormagnetic disks, and memory cards. Software and data used to practiceembodiments of the present invention can be stored on such portablecomputer readable storage media and can be loaded onto persistentstorage 708 via I/O interface(s) 712. I/O interface(s) 712 also connectsto a display 720.

Display 720 provides a mechanism to display data to a user and may be,for example, a computer monitor.

The programs described herein are identified based upon the applicationfor which they are implemented in a specific embodiment of theinvention. However, it should be appreciated that any particular programnomenclature herein is used merely for convenience, and thus theinvention should not be limited to use solely in any specificapplication identified and/or implied by such nomenclature.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the invention.The terminology used herein was chosen to best explain the principles ofthe embodiment, the practical application or technical improvement overtechnologies found in the marketplace, or to enable others of ordinaryskill in the art to understand the embodiments disclosed herein.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the presentinvention. As used herein, the singular forms “a”, “an” and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise.

Each respective figure, in addition to illustrating methods of andfunctionality of the present invention at various stages, alsoillustrates the logic of the method as implemented, in whole or in part,by one or more devices and structures. Such devices and structures areconfigured to (i.e., include one or more components, such as resistors,capacitors, transistors and the like that are connected to enable theperforming of a process) implement the method of merging one or morenon-transactional stores and one or more thread-specific transactionalstores into one or more cache line templates in a store buffer in astore cache. In other words, one or more computer hardware devices canbe created that are configured to implement the method and processesdescribed herein with reference to the Figures and their correspondingdescriptions.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiment, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

Embodiments of the present invention may be used in a variety ofelectronic applications, including but not limited to advanced sensors,memory/data storage, semiconductors, microprocessors and otherapplications.

A resulting device and structure, such as an integrated circuit (IC)chip can be distributed by the fabricator in raw wafer form (that is, asa single wafer that has multiple unpackaged chips), as a bare die, or ina packaged form. In the latter case the chip is mounted in a single chippackage (such as a plastic carrier, with leads that are affixed to amotherboard or other higher level carrier) or in a multichip package(such as a ceramic carrier that has either or both surfaceinterconnections or buried interconnections). In any case the chip isthen integrated with other chips, discrete circuit elements, and/orother signal processing devices as part of either (a) an intermediateproduct, such as a motherboard, or (b) an end product. The end productcan be any product that includes integrated circuit chips, ranging fromtoys and other low-end applications to advanced computer products havinga display, a keyboard or other input device, and a central processor.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present invention has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention. Theembodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

While the invention has been described in detail in connection with onlya limited number of embodiments, it should be readily understood thatthe invention is not limited to such disclosed embodiments. Rather, theinvention can be modified to incorporate any number of variations,alterations, substitutions or equivalent arrangements not heretoforedescribed, but which are commensurate with the spirit and scope of theinvention. Additionally, while various embodiments of the invention havebeen described, it is to be understood that aspects of the invention maybe included by only some of the described embodiments. Accordingly, theinvention is not to be seen as limited by the foregoing description. Areference to an element in the singular is not intended to mean “one andonly one” unless specifically stated, but rather “one or more.” Allstructural and functional equivalents to the elements of the variousembodiments described throughout this disclosure that are known or latercome to be known to those of ordinary skill in the art are expresslyincorporated herein by reference and intended to be encompassed by theinvention. It is therefore to be understood that changes may be made inthe particular embodiments disclosed which are within the scope of thepresent invention as outlined by the appended claims.

What is claimed is:
 1. A method to adaptively pipeline a map stage of aMapReduce job, the method comprising: receiving, by one or moreprocessors, into a first memory buffer, one or more data records from astorage, wherein a size of the first memory buffer is adaptive to one ormore utilizations of one or more resources in a processor, wherein theprocessor is included in the one or more processors; inserting, by theone or more processors, the first memory buffer into a first queue,wherein a size of the first queue is adaptive to one or moreutilizations of one or more resources in the processor; generating, bythe one or more processors, one or more output records from the firstmemory buffer in the first queue by applying a map function to the oneor more data records in the first memory buffer; writing, by the one ormore processors, the one or more output records into a second memorybuffer in a second queue, wherein the size of the second memory bufferand the size of the second queue is adaptive to one or more utilizationsof one or more resources in the processor; deleting, by the one or moreprocessors, the first memory buffer from the first queue; generating, bythe one or more processors, one or more sorted records in the secondmemory buffer by sorting the one or more output records in the secondmemory buffer; responsive to generation, of one or more sorted recordsin the second memory buffer that overflow the second memory buffer,executing, by the one or more processors, a merge and spill of the oneor more sorted records to storage; writing, by the one or moreprocessors, the second memory buffer into a third queue, wherein thesize of the third queue is adaptive to one or more utilizations of oneor more resources in the processor; deleting, by the one or moreprocessors, the second memory buffer from the second queue; merging, bythe one or more processors, one or more sorted records in the secondmemory buffer into an output data file that is stored in storage; anddeleting, by the one or more processors, the second memory buffer fromthe third queue.
 2. The method of claim 1, further comprising,responsive to a reception of one or more data records from a storagethat are compressed into a first memory buffer, decompressing, by theone or more processors, the one or more data records.
 3. The method ofclaim 1, further comprising, a thread scheduler that adaptively appliesa number of threads to executing the map function to increase autilization of the processor.
 4. The method of claim 1, furthercomprising, a thread scheduler that adaptively applies a number ofthreads to generating one or more sorted records to increase autilization of the processor.
 5. The method of claim 1, furthercomprising, compressing, by the one or more processors, the output datafile that is stored in storage.
 6. A method to adaptively pipeline areduce stage of a MapReduce job, the method comprising: receiving, byone or more processors, an input data from a map stage of a MapReducejob into a memory region; inserting, by the one or more processors, theinput data in the memory region into one or more third memory buffers ina fourth queue, wherein a size of a third memory buffer in the one ormore third memory buffers and the size of the fourth queue is adaptiveto one or more utilizations of resources in a processor, wherein theprocessor is included in the one or more processors; generating, by theone or more processors, one or more output records by applying a reducefunction to an input data in a third memory buffer in the fourth queue;responsive to a reception of an input data into a memory region andresponsive to the input data overflowing the memory region, merging theinput data and spilling the input data to storage, wherein the inputdata originated from a map stage of a MapReduce job; inserting, by theone or more processors, the one or more output records into a fourthmemory buffer in a fifth queue, wherein the size of the fourth memorybuffer and the size of the fifth queue is adaptive to one or moreutilizations of the one or more resources in the processor; writing, bythe one or more processors, the one or more output records in a fourthmemory buffer in the fifth queue to storage; and deleting, by the one ormore processors, the fourth memory buffer from the fifth queue.
 7. Themethod of claim 6, further comprising, responsive to a reception of aninput data from a map stage of a MapReduce job, decompressing, by theone or more processors, the input data.
 8. The method of claim 6,further comprising: compressing, by the one or more processors, the oneor more output records in the fourth memory buffer; and writing, by theone or more processors, the one or more records to storage.
 9. Themethod of claim 6, further comprising, a thread scheduler thatadaptively applies a number of threads to execution of the reducefunction such that there is an increase in a utilization of theprocessor.
 10. The method of claim 7, further comprising, a threadscheduler that adaptively applies a number of threads to decompressionof the input data such that there is an increase in a utilization of theprocessor.
 11. A method to adaptively pipeline a reduce stage of aMapReduce job, the method comprising: receiving, by one or moreprocessors, into a first memory buffer, one or more data records from astorage, wherein a size of the first memory buffer is adaptive to one ormore utilizations of one or more resources in a processor, wherein theprocessor is included in the one or more processors; inserting, by theone or more processors, the first memory buffer into a first queue,wherein a size of the first queue is adaptive to one or moreutilizations of one or more resources in the processor; generating, bythe one or more processors, one or more output records from the firstmemory buffer in the first queue by applying a map function to the oneor more data records in the first memory buffer; writing, by the one ormore processors, the one or more output records into a second memorybuffer in a second queue, wherein the size of the second memory bufferand the size of the second queue is adaptive to one or more utilizationsof one or more resources in the processor; deleting, by the one or moreprocessors, the first memory buffer from the first queue; generating, bythe one or more processors, one or more sorted records in the secondmemory buffer by sorting the one or more output records in the secondmemory buffer; responsive to generation of one or more sorted records inthe fourth buffer that overflow the fourth buffer, executing, by the oneor more processors, a merge and spill of the one or more sorted recordsto storage; writing, by the one or more processors, the second memorybuffer into a third queue, wherein the size of the third queue isadaptive to one or more utilizations of one or more resources in theprocessor; deleting, by the one or more processors, the second memorybuffer from the second queue; merging, by the one or more processors,one or more sorted records in the second memory buffer into a map datafile that is stored in storage; deleting, by the one or more processors,the second memory buffer from the third queue; receiving, by the one ormore processors, into a memory region an input data from the map datafile stored in storage; inserting, by the one or more processors, theinput data, included in the memory region, into one or more third memorybuffers in a fourth queue, wherein a size of a third memory buffer inthe one or more third memory buffers and the size of the fourth queue isadaptive to one or more utilizations of resources in the processor;generating, by the one or more processors, one or more output records byapplying a reduce function to an input data in a third memory buffer inthe fourth queue; inserting, by the one or more processors, the one ormore output records into a fourth memory buffer in a fifth queue,wherein the size of the fourth memory buffer and the size of the fifthqueue is adaptive to one or more utilizations of the one or moreresources in the processor; writing, by the one or more processors, theone or more output records in a fourth memory buffer in the fifth queueto storage; and deleting, by the one or more processors, the fourthmemory buffer from the fifth queue.
 12. The method of claim 11, furthercomprising: adaptively scheduling, by the one or more processors, anumber of threads to execute the map function to increase a utilizationof the processor.
 13. The method of claim 11, further comprisingadaptively scheduling, by the one or more processors, a number ofthreads to generate one or more sorted records to increase a utilizationof the processor.
 14. The method of claim 11, further comprising:responsive to a determination that an amount of free storage is lessthan a specified amount, compressing, by the one or more processors, theone or more sorted records.
 15. The method of claim 11, furthercomprising: responsive to i) a reception of an input data from the mapdata file and ii) to the input data overflowing the memory region,merging, by the one or more processors, the input data and spilling theinput data to storage.
 16. The method of claim 11, comprising:compressing, by the one or more processors, the one or more outputrecords in the fourth memory buffer; and writing, by the one or moreprocessors, the one or more records to storage.
 17. The method of claim11, comprising: adaptively scheduling, by the one or more processors, anumber of threads to execute the reduce function such that there is anincrease in a utilization of the processor.
 18. The method of claim 11,comprising: adaptively scheduling, by the one or more processors, anumber of threads to decompress the input data such that there is anincrease in a utilization of the processor.